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ABSTRACT 

Naval Aviation has been tasked to reduce its 1996 human factors related Class A 
flight mishap rate in half by the year 2000. The Human Factors Quality Management 
Board (HFQMB) was established in 1996 to help Naval Aviation achieve its mishap 
reduction goals. Previous HFQMB mishap reduction initiatives focused on flight 
mishaps and aircrew related issues, however a recent shift to broaden the scope now puts 
attention on aircraft maintenance and consequently maintainers. Baker (1998) developed, 
from a Maintainor’s perspective a Climate Assessment Survey (MCAS). The purpose of 
the MCAS was to provide a diagnostic tool to identify potential intervention areas from 
the perspective of maintenance-related mishaps. This thesis seeks to provide a vehicle 
fostering the proliferation of the MCAS throughout the fleet by adapting it for the World 
Wide Web (WWW), and developing a prototype Web site. A usability analysis was 
conducted to form a basis for future modification to the online version of the MCAS, 
with the ultimate goal being an effective tool to promote the reduction of human factors 
related mishaps in Naval Aviation maintenance. 
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I. INTRODUCTION 



A. HUMAN FACTORS QUALITY MANAGEMENT BOARD 
1. Overview 

Following the tragic January 1996 Nashville, Tennessee Navy F-14 mishap that 
killed the aircrew and several local residents, Vice Admiral Bennitt, the then Commander 
of the Naval Air Force, U. S. Pacific Fleet, created a Human Factors Quality Management 
Board (HFQMB) to improve aviation safety through identifying strategies to prevent 
human error. The purpose of the HFQMB simply stated in its charter (1996) is: 

To significantly reduce the rate of occurrence of naval aviation mishaps 
caused by aircrew/operator error by identifying systemic improvements in 
the processes and systems that affect human performance and guard 
against human error. Further, to institutionalize continual improvement in 
these areas. 

Senior Naval Aviation leaders recognized a need to form a coalition between them 
as policy makers and individuals who work directly in the Naval Aviation community. 
Consequently, the HFQMB is composed of a Flag Officer and senior operational Officers 
from both the Navy and Marine Corps including a cross section of all communities and 
ranks from Squadron Commanding Officers to Fleet Lieutenants. The implementation of 
the HFQMB findings was intended to foster a reduction in the 1996 Naval Aviation class 
“A” flight mishap rate by 50 percent within three years and by 75 percent within 10 years 
(HFQMB Charter, 1996). 
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2. Human Factors Quality Management Board Efforts 

The HFQMB’s strategy of taking an across the board approach to improve its 
processes and systems encompassed a three-prong approach, including: (1) Mishap Data 
analysis, (2) Benchmarking Best Procedures, (3) Climate Safety Assessment (See Figure 
1). Mishap Data Analysis included the practice of observing patterns that may have 
formed in past mishaps, and trends that may been have established in relation to human 
factors. Bench Marking involved observing the practices used in other organizations to 
combat human factor related error. The Climate Safety Assessment was designed to 
qualify issues that might impact these human factor errors. 



Mishap Data 
Analysis 



Benchmarking 



Climate Safety 
Assessment 





Brainstorming 



► 

► 



Recommendations 
for Intervention 



Figure 1 . HFQMB Methodology 
From “AIRPAC Brief,” by CDR J. Schmidt, March 1998. 



The first area of interest to the HFQMB, analysis of mishap data, represents 

I 

research conducted without the need for external data collection. The Naval Safety 
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Center (NSC) maintains a database of mishaps dating back 20 years; this historical data is 
categorized by 1050 characteristics (NSC, 1999). Taking the NSC data and classifying 
the cause and type of mishaps enabled aggregate analysis by the HFQMB. The NSC’s 
mishap database also allowed the HFQMB to analyze the mishaps with enough specificity 
that when coupled with the appropriate modeling techniques, enabled meaningful action 
development. 

Benchmarking processes/practices was the second area of endeavor by the 
HFQMB for mishap reduction. Benchmarking is the process through which internal or' 
external organizations' performance levels are compared, and its processes/practices are 
observed. The organizational benchmarks can be utilized to establish guidelines and 
ultimately lead to "Best Practice" decisions (Chapman, 1997). In this instance, the 
HFQMB observed processes from organizations that seek the same safety goals as Naval 
-Aviation such as, the U.S. Air Force, and the commercial airlines, etc. (HFQMB, 1996). 

The third area of focus by the HFQMB is addressed by the Command Safety 
Assessment (CSA), a tool designed by Ciavarelli and Figlock (1997) at the Naval 
Postgraduate School to measure safety climate in a squadron from the aircrew 
perspective. The CSA (Discussed in subsequent chapters) enabled the HFQMB to 
measure the climate and process effectiveness within an organizations’ flight operations. 
Recommendations were deemed a critical part of the overall success of the HFQMB’s 
actions. Using the information acquired through the means of observation and collection, 
the HFQMB prioritized the recommendations in terms of their potential to achieve the 
mishap reduction goals and forwarded them to the appropriate authority for coordination 
and implementation at the organizational level of Naval Aviation (HFQMB, 1996). 
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3. Human Factors Quality Management Board Results 

Historically, the mishap rates of the last two decades have been low by past 
standards (See Figure 2). However, despite a low mishap rate, it is important to realize 
that Naval Aviation strikes several aircraft per year. An example of the effect lost aircraft 
have on the fleet is illustrated by the data taken from fiscal year 1997 through 1998. In 
FY ‘97, a total of 65 aircraft were accepted into the Naval Aviation inventory (Naval 
Aviation News, 1998). During FY ‘98, 36 aircraft were removed from the inventory as a 
result of mishaps, (NSC, 1999). The aircraft lost in 1998 represent 55 percent of those 
purchased in the previous year. The class “A” mishap rate has been essentially on a 
plateau since 1988. Without a change in leadership action or the conduct of Naval 
Aviation, the mishap rate would not be expected to deviate from its current level. 

Contributing to mishap data analysis, Shappell and Weigmann (1995) adapted 
Reason’s (1990) Human Error Model to re-classify the divisions of Naval Aircrew Flight 
mishaps. This classification model took the larger category of human error and expanded 
it to three sub-areas of responsibility (Nutwell & Sherman, 1997). The new divisions of 
human error consisted of; unsafe aircrew conditions, unsafe supervisory conditions, and 
unsafe aircrew acts. The redesigned model gave a better understanding of the human 
cause factors than previous classifications in the traditional model; more accurate 
reporting resulted in improved understanding, and clearly indicated the appropriate 
intervention areas to achieve the goals sought by the HFQMB. Notably, in a report issued 
in 1997 by the Department of the Navy, 40 percent of the procedure violations committed 
by aircrew were conscious breeches, and 50 percent of the supervisory failures were 
labeled unsafe supervisory conditions. Both aircrew and supervisory conditions are an 
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indication of climate within the organization. Relatively little work has been completed 
on organizational factors when considering the effectiveness of a squadrons performance 
and safety (Ciavarelli & Figlock, 1997). Program and policy of the squadron are a part 
the safety climate and culture maintained within the organization. 



Figure 2. FY 50-99 Flight Mishap Rates. 

NAVAL AVIATION MISHAP RATE 




From: Naval Safety Center 1999. (Through 30 JUN 99) 



The efforts of the HFQMB were expected to bring about fundamental changes in 
the way business was conducted in Naval Aviation, with these changes, the established 
mindset regarding attainable mishap reduction would have to change. (HFQMB Charter, 
1996). The recommendations of the HFQMB resulted in a record mishap rate for the 
U.S. Navy in 1997, with a mishap rate of 1.29 %, the lowest level of class A mishaps ever 
(NSC, 1999). The U. S. Marine Corps reached a comparable milestone in 1998 with the 
organizations lowest mishap rate of 2.52 % (NSC, 1999). The past actions taken by the 
HFQMB have had the desired effect of reducing the human factor related mishap rate. To 
continue to reduce the human factors related mishaps, refinements in the process 
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evaluation would need to take place. Nutwell and Sherman (1997) state human error 
represents a large portion of the causal factors that contribute to Class “A” mishaps and 
propose additional reductions in the mishap rate could be attained with refinements in this 
area. Traditional approaches targeting aircrew human factors should be expanded to 
include the maintainer with potential gains in mishap reduction. If further improvements 
in mishap reduction were to occur continued exploration of all facets of prevention 
needed to be studied and understood. Efforts in mishap reduction were concentrated in 
the aircrew side of human factors caused mishaps, expanding the focus to include the 
maintenance caused human factors mishaps could enhance the overall reduction of the 
mishap rate. The desire to measure the climate of safety from a maintainer’s viewpoint 
was a topic that necessitated the development of a tool for this purpose. 



B. AVIATION SAFETY CLIMATE ASSESSMENT 
1. Command Safety Assessment 

The Command Safety Assessment (CSA) was developed by Ciavarelli and 
Figlock (1997) of the Naval Postgraduate School; its goal was to estimate the 
effectiveness of organizational safety programs. To assess the effectiveness of safety 
programs and processes, it is necessary to understand the climate that is maintained 
within the organization. Organizational culture is defined by Merritt and Helmreich 
(1996) in the following marmer. 
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Culture can be defined as the values, beliefs, rituals, symbols and 
behaviors that we share with others that help define us as a group, 
especially in relation to other groups. Culture gives us cues and clues on 
how to behave in normal and novel situations, thereby making the world 
less uncertain and more predictable for us. There are two important and 
distinct components of culture. The surface structure, or outer layer of 
culture consists of observable behaviors and recognizable physical 
manifestations such as member’s uniforms, signs and logos, and 
documents. The deep structure, or inner layer of culture, consists of the 
values, beliefs and assumptions which underlie the surface structure and 
provide the logic which guides the members’ behaviors. 



Measuring culture and looking at the safety climate an organization fosters is part 
of understanding the organization and the people that make up the group. Each person 
brings their history and multi-cultural memberships with them to an organization (Merritt 
& Helmriech, 1996). If organizational culture has an impact on the safety of the group, it 
would be desirable for the group to have a unified, integrated organizational culture, 
rather than one made up of diverse values. Much attention has been focused on human 
error in complex systems, but relatively little has been accomplished on the potential 
contribution that organizational factors have on crew performance and safety (Ciavarelli 
&Figlock, 1997). 

Work in the area of climate measurement within Naval Aviation squadrons is 
built on research conducted by Roberts (1990). Roberts discussed the attributes in 
organizations that were successful in reducing risks associated with hazardous operations. 
Her label for this type of organization was “high-reliability” and described an 
organization that worked in a hazardous environment, yet produced very low rates of 
accidents. The CSA study was based on a model of high reliability organizations, and 
produced a questionnaire that would provide feedback to the HFQMB concerning the 
issues of command climate, morale of Naval personnel, workload, resource availability. 
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and estimated success in safety intervention programs. This newly developed tool, the 
CSA, could be used by the Aviation Safety Officer (ASO) to assess the organizations’ 
effectiveness regarding their commands safe operation (Nutwell & Sherman, 1997). 

3. Maintenance Climate Assessment Survey 

Safety culture within an organization effects the entire output of the group. Naval 
Aviation is recognized as a high-risk organization, devastating consequences occur if 
something goes wrong within any aspect of the organizations operation. Just as the 
concept of the CSA was created as an extension of the principles guiding the study of 
high reliability organizations (Ciavarelli & Figlock, 1997), the Maintenance Climate 
Assessment Survey (MCAS) became the maintenance focused extension of the CSA 
(Baker, 1998). The MCAS was developed with the notion that operational climate and 
maintenance climate hold the same importance within the unit, therefore, both contribute 
to safe completion of the organizations mission. With maintenance actions contributing to 
the mishap rate, the need to modify a survey tool to address the issues pertaining to 
maintenance became relevant (Baker, 1998). 

The MCAS has a focus on safety climate from the aviation maintainor's 
viewpoint. Bakers (1998) goal was to develop, administer, and validate a questionnaire 
to assess the effectiveness of Naval Aviation maintenance operations in the management 
of risk. Baker concludes that the developed prototype MCAS with a few noted 
exceptions conformed to the existing Model of Organizational Safety Effectiveness 
(MOSE) quite readily, he successfully adapted the model to make it maintenance specific. 
Statistical analysis allowed the original 67 question MCAS to be reduced to a more 
practical length of 35 questions. Refinement and consideration for distribution of the 



8 



MCAS offer areas of further development. 



C. RESEARCH OBJECTIVE 

The objective of this study is to further enhance the effectiveness of the MCAS 
developed by Baker at the Naval Postgraduate School (Baker, 1998). This study creates 
an online version of the MCAS, and incorporates this version of the MCAS within an 
electronic distribution system and tests the usability of the online system. The intent is to 
increase the availability of the MCAS, enhancing present efforts to reduce maintenance 
related mishaps and the overall mishap rate. 

D. STATEMENT OF PROBLEM 

Utilizing the distribution capability that is created by the World Wide Web, an 
online version of the MCAS can be distributed to an ever-increasing audience with 
virtually no increase in distribution expense (Lewis, 1998). To further reduce the 
maintenance attributed aviation mishaps, safety process and programs that contribute to 
safe operation will need to improve. Not simply seeking improvements in process and 
programs, the aviation community sought the invention of new processes to enable the 
desired improvements in mishap reduction (Nutwell & Sherman, 1997). 

Distribution of information on the World Wide Web offers the opportunity for the 
Naval Aviation community to '‘invent” the new process that it desires, bringing about 
changes in safety and ultimately reducing the mishap rate. As Lew'is (1997) states, the 
World Wide Web offers increased exposure to an audience without increases in 
distribution costs, and as a distribution medium, the World Wide Web can disseminate 
information that was originally designed for distribution in other forms. Specifically, the 
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MCAS was created to be distributed in paper form, with modification to format, the 
MCAS can be reconfigured to World Wide Web specifications. Distribution of the 
developed MCAS by an effective Web based method will enable more organizations to 
use this tool for self-evaluation. 

The main thrust of this study focuses on the effective reconfiguration and 
distribution of the existing MCAS through the World Wide Web. Usability of the online 
version of the MCAS was the measurement of effectiveness. Understanding and 
measuring the effectiveness of the safety program is vital to the reduction of human factor 
caused mishaps. It is essential to provide tools to the aviation organization for self- 
assessment of the effectiveness of their processes and programs. Four issues are relevant 
in the discussion of an electronic distribution of the MCAS, specifically, this thesis 
explores the following questions: 

1. Is there a need to disseminate and administer Web based materials to aviation 
units? 

2. Do U.S. Navy and Marine Corps squadrons have the necessary 
software/hardware capabilities to utilize the World Wide Web? 

3. Can the MCAS be effectively distributed and administered using the World 
Wide Web? 

4. Once distributed, do personnel have the necessary ability to utilize the 
electronic version of the MCAS? 

E. SCOPE AND LIMITATIONS 

The intent of this study is to gain a better understanding of the distribution 
capability the World Wide Web offers organizations with the proper hardware and 
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software in place. It is essential to determine the usability of an online version of the 
MCAS for use by Naval Aviation squadrons. Subjects from the Naval Aviation School 
of Safety will be used due to their close proximity, this effort represents a preliminary 
effort to measure the usability of the online version of the MCAS. An exit survey will 
allow the users to rate their impression of this method of MCAS distribution. 
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II. LITERATURE REVIEW 



A. OVERVIEW 

The review of the literature considers three areas pertinent to the effective online 
delivery of an organizational climate assessment tool. Areas covered include: 

1) Organizational climate and its assessment to provide background on the traditional 
process; 2) High Reliability Organizations (HRO) as a model for effective organizational 
climate is considered and its application to the military; 3) Development of the Command 
Safety Assessment (CSA) and the Maintenance Climate Assessment Survey (MCAS) for 
Naval Aviation based on the HRO model; 4) The software development process entailed 
for organizational assessment tool delivery; and 5) The Human computer interface 
concept, functionality and usability. The information is summarized and considered in 
context of the research questions. 

B. ORGANIZATIONAL CLIMATE AND ASSESSMENT 

Daft (1989) defines organizational climate as: 

. ..the set of key values, guiding beliefs, and understandings that are shared 
by members of an organization. It defines basic organizational values, and 
communicates to new members the correct way to think and act, and how 
things ought to be done. It represents the unwritten, feeling part of the 
organization, (p. 503) 

Climate is the unseen part of an organization that helps members make sense of 
organizational events and activities (Daft, 1989). When an organization enjoys 
widespread agreement, climate within the organization is strong, conversely, if little 
agreement exists within the organization, climate is weak. Daft (1989) also notes that 
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leadership plays a large role in the organizational climate. He states that simple mandates 
of policy do not form an organizations cultural climate, values and beliefs come from 
observing leaders within the organization. Climate is an important feature of the 
organization, and if a change in organizational climate is desired, leadership must 
implement more than simple policy to effect the needed changes (Schein, 1992). 

To understand more fully the process of change, it is helpful to study the needed 
elements that will successfully facilitate it. Burke (1994) looks at organizational 
development as it relates to change. Much work in this area is built on a model 
developed by Lewin (1958) called the three-step procedure of change. The first step of 
this model is unfreezing the present level of behavior, which is no more than reducing 
prejudice and clarifying the area that requires change. The second step is movement, take 
action that will change the social system from its original state of behavior. The final and 
third step is refreeze, or establish processes that will secure the new behavior from 
change. 

Schein (1992) elaborates on Lewin’ s model, enlarging and defining the scope of 
each step. He further sets forth useful assumptions when studying an organization and 
attempting to measure culture. Schein notes, it is important to keep in mind that the 
contextual meaning of cultural assumptions can only be fully understood by the members 
of that culture, so creating a vehicle that helps the members of the group understand their 
own culture is actually more important than the researcher gaining this perspective. Since 
culture is a shared set of assumptions, obtaining initial data in a group setting is 
appropriate. 
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C. HIGH RELIABILITY ORGANIZATION (HRO) 

In research conducted by Roberts (1990), and extended by Libuser (1994), an 
HRO is defined as an organization that operates in a hazardous environment, yet produces 
very low rates of accidents and incidents. HROs seem to have less than their “fair share” 
of failures despite operating in a complex, demanding environment involving significant 
risks and hazards. Roberts and Libuser note that these HROs had certain characteristics 
that made them organizationally effective managers of risk. Libuser (1994) went on to 
develop a Model of Organizational Safety Effectiveness (MOSE) that categorized five 
traits that were common among this type of organization, these traits include: process 
auditing, reward systems, quality of operations, risk perception and command control. 

HROs are found in many areas of manufacturing and everyday life (Perrow, 
1984). The operation of power plants, oil refineries and transportation represent a few 
industries where accidents are rare but if they occur have catastrophic and far reaching 
effects (Reason, 1998). The operating concepts of HROs can apply to many other 
industries where accidents can not be tolerated, the food industry, drug manufacturers, 
financial institutions are just a few of the organizations that characterize groups that 
depend on high reliability. Finally, the HRO concept can also be extended to military 
applications (Roberts, 1988). 

Reason (1998) describes a theory of active and latent causes of accidents and 
applies these causal factors to HROs. An active failure is one that is directly attributable 
to an action or violation by an individual directly influencing a system. Latent conditions 
may exist for years within a system and not cause an accident until the proper sequence of 
events come together simultaneously. He contends the perception of individuals within 
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an HRO is that iron clad defenses are in place preventing accidents, however in practice, 
organizational defenses are not iron clad, and may have certain weaknesses. When the 
weaknesses in separate defenses occur together in the proper sequence, failure may result 
(See Figure 3). Meeting peak requirements or time pressures may cause individuals 
within the HRO to neglect the ongoing process of finding holes in the accident prevention 
strategy; infrequent mishap occurrence contributes to this mindset (Reason, 1998). 




Figure 3. Perceived and Actual Defenses from Risk 

Research conducted by Roberts (1990) and Libuser (1994) indicate that the 
complexities of systems today necessitate new strategies for prevention, the simple direct 
causation of an accident by one individual does not fit a HRO. HROs must continually 
seek new methods of process improvement and not become complacent despite the fact 
they suffer from few accidents (Perrow, 1984). The perception that these organizations 
maintain “iron clad” defenses against hazards it can not hamper continual process 
improvement (Reason, 1998). Strategies to achieve a HRO would include a strong 
hazard reporting system with no fear of retribution by the members, as well as 
maintaining a flexible, learning culture within the organization. 
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Bierly and Spender (1995) adapt the HRO model concept for military application 
and claim that a nuclear powered submarine fits many of its assumptions. They observe 
culture interacts with the formal organization, and that the strong organizational culture 
within the nuclear submarine supports the formal structure. Roberts (1988) finds an 
increasing number of organizations in modem society fit the HRO model, observing the 
size and complexity of the aircraft carrier organization coupled with a seemingly low 
incidence history in dealing with high risk evolutions qualified it as a HRO. Baker (1998) 
states, “In high-risk endeavors such as aircraft operations, it is essential that such support 
be flawless, as the price of deviations is high.” Naval Aviation as a whole with its 
complexity, high risk, and low tolerance for error fits the criteria to be a HRO. 

Building on the work of Roberts and Libuser, Ciavarelli and Figlock (1997) 
tailored the MOSE to fit the conditions found in Naval Aviation. They created the CSA, 
a tool designed to assess the safety climate within a Naval Aviation squadron from an 
aircrew perspective. The goal of this research was to develop and apply a methodology 
that enabled a command to assess it performance on key safety issues, providing a 
feedback tool as part of the mishap prevention process. Planned follow on efforts include 
cross-comparing squadrons, aircraft type and demographic variables in an attempt to 
construct an empirical foundation for assessing hazards and risk. 

Baker (1998) furthered the efforts of the CSA by designing and validating the 
MCAS that would accomplish the same objectives as the CSA, but from the perspective 
of the aviation maintainer. With the help of maintenance experts, he took a list of 
maintenance topics and created 67 questions specifically tailored to aviation maintenance. 
Baker’s initial 67 question survey through factor analysis was further refined and reduced 
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to a 35 question instrument. The MCAS was successfully proven to support each of the 
MOSE components. 

Currently, Naval Postgraduate School Operations Research Department students 
are engaged in taking the MCAS and validating it by analyzing the results within an 
aircraft community, and across aircraft communities (B. Goodrum, Personal 
Communication . 1999; T. Oneto. Personal Communication . 1999). To further enhance 
the effectiveness of the MCAS, this study seeks to provide a wide reaching delivery 
system on the World Wide Web. Effective distribution will involve the creation of ah 
electronic delivery system, and the usability testing of that system. 

D. SOFTWARE TOOLS 

The software necessary to carry out the successful development and 
administration of the online MCAS includes two categories: developmental software, and 
user software (Alexander & Tate, 1999). The developmental software in this research 
created both the Web site, and also the online version of the MCAS. The user software is 
the tool each user must employ to utilize the online MCAS once developed; this software 
is determined by the end user and is a World Wide Web browser. Both categories of 
software and its employment are discussed below. 

1. Developmental Software 

Perseus Survey Solutions for the Web is the software that was used for the 
transformation of the MCAS to an online version of the same. The Perseus Corporation 
makes this software product Microsoft Office compatible (Strebe & Perkins, 1998). 
Compatibility with Microsoft products allows for future expansion of the online MCAS 
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to include functions contained in Microsoft Access and Microsoft Excel. 

The most common application of this development software is construction and 
distribution of surveys, but its other functionality includes the development of inquiry 
forms, feedback questionnaires and order forms (Perseus Survey Solutions User Guide, 
1998). Functioning as a word processor. Survey Solutions for the Web acts as an 
interface between the plain language text formulation and translation to Hypertext 
Markup Language (HTML). Once textual data is converted to HTML, it may be posted 
on a server and users may access the stored data through their computer and an Internet 
browser. The storage and availability of these files is further described in the discussion 
of the World Wide Web. 

Data collection is the final function that the Perseus software package carries out 
as part of the design of the online MCAS. Once collected through the user response to 
questions within the online MCAS, data is packaged in a standard tab separated form 
(ASCII TSV) which may be further stored, or interpreted using Microsoft Access or any 
program that allows the ASCII TSV data entry (Perseus Survey Solutions User Guide, 
1998). 

The second software development tool used was Netscape Navigator Gold 
Version 4.5. This software package was used in the development of the Web site that 
presented the online MCAS to the user. Netscape Navigator Gold utilizes a powerful 
What- You-See-Is- What- You-Get (WYSIWYG) HTML generator that uses many of the 
same functions found in word processors (Authoring Guide, Netscape Navigator Gold, 
1995). Maintenance Aviation Safety material was gathered from existing sources and 
combined with original material to form a distribution vehicle for the MCAS. 
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2. User Software 



The final piece of software used in this project is the Internet Browser (browser). 

A browser is defined as “a computer program that enables the user to read HYPER-TEXT 
from disk files or on the World Wide Web” (Downing, Covington & Covington, 1 996). 

A browser is the software each end user must utilize to access the online MCAS, and 
becomes the most dynamic portion of the all the software packages used. The dynamic 
nature of Internet Browsers is discussed more fully in subsequent sections and will be an 
ongoing concern in a Web based project. 

E. WORLD WIDE WEB 

1. Overview of the World Wide Web 

The World Wide Web is a unified, interconnected interface to the vast amount of 
information stored on the computers around the world (Sobell, 1995). Information can be 
displayed on the World Wide Web using a combination of text, graphics, animations and 
audio to form dynamic documents. Once created, these documents are displayed on a 
user computer by an Internet browser. A browser provides easy access to Web sites that 
are posted on many servers connected to the World Wide Web. A series of Clients, 
computers that receive information from other computers, and Servers, a computer that 
provides services to another computer, make up the Internet. 

Composed of HTML code, a Web site is made up of one or more pages to create 
an organized presentation of material through the use of Hyperlinks. The Authoring 
Guide, Netscape Navigator Gold (1995) defines the Web site as a package that contains; 
topics that can be linked to from other Web sites, may require the use to scroll the browse • 
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window to see the entire page, can contain links to graphic files and other pages, and 
finally can be located on a server so that other users may access the page. Below, Figure 2 
illustrates the composition of a Web site that is made up of several Web pages, connected 
by use of Hyperlinks. 

Hyperlinks are the mechanism that allow an entire Web site composed of many 
Web pages to be connected in a logical sequence. The use of Hyperlinks to tie together 
multiple Web pages of the same theme allow a user to navigate a rather complex Web site 
with ease and little confusion. The World Wide Web offers and opportunity to distribute 
information to a vast audience at very little cost (Lewis, 1998). 




WebSite 



2. The Internet Browser 

The browser is an integral part of using any online software applications, 
including the online MCAS. Each browser interprets HTML code by applying a set of 
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defaults established by the individual user. A default is a set of assumptions that a 
browser makes unless it is given instructions to the contrary (Downing, Covington & 
Covington, 1996). These defaults within the browser determine how the browser 
interprets the HTML code and displays the information on the users monitor. Different 
interpretations of HTML code result in different views of the same information. For this 
reason, two browsers may interpret the same HTML code differently. In the design of 
Web sites, it becomes necessary to consider the different default settings users will utilize 
and anticipate the effects this may have on data within a Web site application (Randall, 
1999). 



3. The Client/Server Model 

The Client/Server model is the basic model that enables individual users to access 
a Web site at any given time (Sobell, 1995). The Server is a program that runs all the 
time waiting to receive a request for the information it contains in its data stores or 
memory. World Wide Web clients are the computer users who access a server through 
the use of a browser. The Client/Server interaction occurs using a computer language 
known as Hypertext Transfer Protocol (HTTP). HTTP is a computer language that works 
across all platforms; consequently, whether the Client and Server are the same type of 
computer does not inhibit the ability for the two computers to communicate (Downing, 
Covington & Covington, 1996). When the Client requests information from the Server 
via HTTP, the request is acknowledged and data is sent back to the Client. (See Figure 5) 
The Client/Server Model is responsible for enabling the efficiencies in 
distribution of online material throughout the World Wide Web (Tanenbaum, 1992). 
Once posted on a Server, the Online MCAS will incur the same distribution cost as it 
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reaches an ever-increasing audience. The efficiency in online distribution is analogous to 
the infinite shelf space retailer of the fnctionless economy (Lewis, 1997). The ability to 
distribute training materials at a small, fixed cost, to an audience that is potentially 
infinite represents a method of improvement for training that may facilitate reaching 
mishap reduction goals set within the organization of Naval Aviation. 





WWW 

CLIENT 



Request for Data 
using HTTP 



Answers request for data 
using HTTP 




SERVER 




Client 



Figure 5. Illustration of the Client/Server Model 



F. HUMAN USABILITY ISSUES 

Hawkins (1998) observes that “Human factors is about people... It is about their 
relationship with machines and equipment, with procedures and with the environment 
about them.” He illustrates this relationship through the SHEL model, where the elements 
of Software (e.g., training, procedures, etc.). Hardware (e.g., controls, displays, etc.). 
Environment (e.g., temperature, lighting, etc.), and Liveware (e.g., operator, systems, 
etc.), pair to form a human system interface. Liveware is considered the hub of the model 
and all remaining components must be effectively matched to this central component. 
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Human factors principles have traditionally been applied to hardware devices such as 
monitors, keyboards, etc. as well as workstation design to optimize their fit with human 
needs and capabilities (Grandjean, 1987). The proliferation of hardware and software 
technology necessitated the same intensive application of human factors technology to 
enhance software Human Computer Interface (HCI) design (Brown, 1989). Today, most 
software and design applications integrate HCI design concepts including; fonts, color, 
etc., as well as screen layout and functionality. 

The “browsers” that the World Wide Web is accessed through have gone passed 
many HCI design iterations to maximize their efficiency and ease of use (Frederickson- 
Mele, Levi, & Conrad, 1999). Web software and that used in Web site development have 
also been crafted using HCI concepts with similar objectives as in software development. 
Using the Internet as a distribution method raises unique design issues that are dependent 
on user terminals and pre-selected defaults, consequently. World Wide Web applications 
benefit from the HCI research in the browser they are viewed through. This unique set of 
user circumstances requires that Web sites be tested for usability by their intended 
audience. 

1. Usability Testing 

Software and its design presents many challenges to the user in trying to 
accomplish a given task. A reasonable goal of any software project should be the ease of 
use of the final product (Jordan, 1 998). Usability issues are not only important within the 
realm of consumer products, but also those products used within a commercial or 
professional context (Jordan, 1998). Past studies indicate, with the exception of safety 
testing, usability testing takes place in the worst possible arena, the marketplace (Randall, 
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1999). In other words, Randall (1998), states whether a product was usable or not was 
left up to the end user, after the purchase or attempted use of the final product. Formal 
usability testing became recognized about a dozen years ago in the computer software 
industry. Usability testing does not speed the design process up, but if user satisfaction is 
the goal, the increased development time is well spent. Constructing a site to meet the 
needs of your intended audience increases the impact the overall presentation has upon 
the user (Frederickson-Mele, Levi, & Conrad, 1999). 

Delivering a prototype of the product to a test group followed by a questionnaire 
is a method Jordan (1998) found beneficial in determining the attitudes and expectations 
of potential product users. Jordan specifies that data obtained from usability studies 
should answer the following questions: 

• Are there usability faults in the product? If so, what are these faults? 

• How can these usability faults be solved? 

• What are user attitudes towards the use of this product? 

• What design changes are required to make the users' attitudes more positive? 

Open ended questions, allow the respondent to write their own answers to questions and 
prove particularly useful in determining what the important issues were to the users. A 
usability study enhances the effectiveness of modification to a prototype, ultimately; the 
final product will have more utility to the end user (Jordan, 1 998). 

2. Writing For the Web 

When writing for Web sites, studies have shown that users do not actually read: 
instead, they scan the text (Nielsen, 1994). This finding suggests a style of writing that 
can be absorbed through scanning rather than through in-depth study. During Nielsen’s 
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four years of usability testing, he considered site architecture, navigation, search, page 
design, and style yielding surprising results. When asked to comment on the Web site, 
users critiqued content quality and relevance to a much greater extent than navigational 
issues or page elements (Nielsen & Sano 1994; Nielsen, 1997). This suggested content 
would play a very large part in the users perception of the Web site. Morkes and Nielsen 
(1997) suggest three main content oriented conclusions be drawn from their study. 

1 . Users do not read on the Web; instead they scan the pages, trying to pick out a few 
sentences or even parts of sentences to get the information they want. 

2. Users do not like long, scrolling pages: they prefer the text to be short and to the 
point. 

3. Users detest anything that seems like marketing fluff or overly hyped language 
and prefer factual information. 

It is clear that users require concise presentations of content when on the Web, and have 
very low tolerance for superfluous words or ideas in the material. 

Morkes and Nielsen (1997) state that writing effectively for the Web is a function 
of sensitivity to the users needs balanced with the message you wish to covey to the user. 
Writing in a concise, scannable style would seem to fit the users need for quick fact 
finding, while ensuring content was available to the user. It was noted that when a page 
comes up, the readers attention shifts to center screen and the body of the text was read 
before they looked over headers or other navigational elements that made up the page 
(Morkes & Nielsen, 1997). Similar findings on the importance of content resulted from 
the work of other similar studies of online materials (Spool, Scanlon, Schroeder, Snyder, 
& DeAngelo, 1997). This indicates the need to maintain a concise writing style on the 
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Web page, paying particular attention to the relevance of the material presented to the 
intended user. 

G. SUMMARY 

An understanding of organizational climate leads research in the direction of 
categorizing Naval Aviation as a HRO. As a HRO, Naval Aviation safety programs and 
processes can be analyzed and improved using the CSA and MCAS. Working within the 
confines of current distribution methods, the WWW offers unique opportunities for 
widespread, inexpensive distribution of safety material. Special consideration must be 
given to design issues regarding usability of software applications. A vehicle to distribute 
the online version of the MCAS will further the need to assess safety program 
effectiveness in Naval Aviation. 
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III. METHODS 



A. RESEARCH APPROACH 

Naval Aviation seeks methods to improve its process and programs regarding 
safety in an effort to reduce mishap rates. The MCAS developed by Baker (1998) was 
designed to be utilized by squadrons to assess the effectiveness of their safety programs 
from a maintainers perspective, making it possible to measure the safety climate within 
an aviation command. Making the MCAS available to more aviation units through 
effective distribution on the World Wide Web is the goal of this study. 

To measure the effectiveness of the online MCAS, a prototype Web site was 
developed and distributed to a sample user group. Each subject tested the Naval Aviation 
Maintenance Safety Web site with a prepared task list that required them to navigate 
through the pages of the Web site. At the completion of the task list, the subject would 
have viewed all the pages of the Web site, and presumably formed an opinion on its 
effectiveness. Subjects upon completing the task list filled out an exit survey composed 
of demographic questions of their computer background and perusal of the protot>pe Web 
site. 

B. DATA COLLECTION 

1. Subjects 

Students attending the Aviation Safety Officer Course (ASO) at the School of 
Aviation Safety served as subjects in the study. Selection to the ASO course is made by 
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individual Aviation Commands cind yields a cross section of the Naval Aviators and 
Naval Flight Officers from all aircraft communities within the U.S. Navy and Marine 
Corps. Squadron ASOs are responsible for the safety programs in all squadrons and will 
likely be one of the primary users. Each subject from this test group will have a direct 
influence on the safety climate within their squadrons. 

2. Apparatus 

The Naval Postgraduate School’s School of Aviation Safety provides a computer 
lab for the ASO students during their training. Each Officer is issued a password that 
allows computer usage through a group account. The computers in the lab each had a full 
functioning prototype of the Aviation Maintenance Safety Web Site. After a subject 
gained access to the computer through use of the group account, the icon “Aviation 
Maintenance Safety Web site” resided on the computer desktop. Once the Web site was 
opened, it appeared as any World Wide Web site, although this prototype was installed on 
the hard drive of the computer rather than on a server on the Internet. The prototype was 
kept locally on the lab computers to facilitate changes that would occur in its 
development. Microsoft Internet Explorer was the user software provided to the subjects. 
All of the functions within Internet Explorer operated in the same manner as if the 
application was taken from the World Wide Web. 

3. Instrument 

The exit survey consisted of three parts: 1) Demographic items or computer 
experience, 2) Likert type questions assessing feeling towards the Web site, and 3) Open- 
ended items to elicit subjective opinions. The first part, collecting demographic 
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information, allowed the subject to choose from a list of descriptors (e.g. Squadron level. 
Intermediate level, etc.). The second and third part of the survey was a combination of 
Likert style questions and open-ended questions that allowed the subjects to inteiject 
comments they felt were relevant. The Likert questions were constructed using a five 
point rating scale with verbal anchors, as follows: Strongly Agree, Agree, Neutral, 
Disagree, and Strongly Disagree. The open-ended questions were included in the 
questionnaire to elicit respondent viewpoints regarding such factors as the overall 
impression of the Web based distribution, and comments regarding improving the 
usability of the Web site. The open-ended questions provided subjects an opportunity to 
comment on aspects of the prototype Web site they felt were not adequately covered in 
the previous portions of the survey (See Appendix D for survey). 

4. Procedure 

The subjects were divided into two groups of 26 and presented with an overview 
presentation of the study. The overview consisted of a projected computer demonstration 
of the Aviation Maintenance Safety Web site, and distribution of the materials necessary 
to carry out the user test. The materials used to direct the subjects exposure to the Web 
site included: Instructions for viewing the Aviation Maintenance Web site (Appendix B); 
Aviation Maintenance Safety Web Site Task List (Appendix C): and User’s Impression of 
the Maintenance Safety Web Site and MCAS (Appendix D). The Instructions for 
viewing the Aviation Maintenance Web Site was a guide for the subject to follow 
enabling him/her to log on to the group account provided in the computer lab and access 
the Web site installed on the computer. As a portion of the group overview, specific 
questions subjects expressed on using the computers in the lab were addressed. The Task 
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list was designed to take the subject through a series of planned navigation routes within 
the Web site. By completing the task list, the subject would have navigated through the 
entire prototype of the Aviation Maintenance Safety Web site. 

C. DATA TABULATION 

The data was transcribed from the exit survey into the software program SPSS 
(SPSS V. 9.0). Multiple choice responses were each assigned a value from one through 
four. The Likert style questions were based on a five point scale, these questions were 
coded into the SPSS software using five (5) through one (1) corresponding to the 
respective anchors (Strongly Agree, Agree, Neutral, Disagree, and Strongly Disagree). 
Once the data was transcribed, the analysis tools in the software SPSS were used to 
generate descriptive statistics including the mean, standard deviation, range, and 
frequency distribution. Content analysis was conducted on the responses provided from 
the open-ended survey questions. 

D. DATA ANALYSIS 

1. Analysis Strategy 

The data from the surveys was transferred to the statistical analysis program SPSS 
(SPSS v.9.0) for analysis and interpretation. One-way Analysis of Variance (ANOVA) 
testing matched the level of user experience. Independent Variable (IV), with selected 
Dependent Variables (DV) gathered in the survey. The one-way ANOVA was used to 
assist in answering the difference (group comparison) issues raised from the study. 
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IV. RESULTS 



A. FREQUENCY AND CONTENT ANALYSIS 

The 21 item exit survey was administered to 52 subjects with a response rate of 
88% (n = 46). The test users represented a cross section of the aviation commands that 
make up the squadrons of the Fleet. Each aviation command is required to have an 
officer trained by the School of Aviation Safety; as a result, the entire fleet is represented 
in this school at any given time. The sample included flight qualified Officers from Naval 
and Marine Corps Aviation and did not include Non-Commissioned Officers or Enlisted 
personnel. 

1. Demographic Information 

The material collected in Part I, the demographic portion of the exit survey 
determined the experience level each subject had with computers, as well as, the Internet 
and Internet browsers. This information is later used to determine if experience level 
with Internet browsers affected a subject’s level of satisfaction, and impacted the usability 
of the Web site. Question one on the survey revealed that all participants were members 
of aviation units that performed maintenance at the squadron level (n = 46, 100%). 
Question two asked, “How long have you been using an Internet browser?” Table 1 
shows the breakdown regarding experience each subject had with Internet browsers. A 
total of 82 percent (n = 38) of the subjects had at least one year of experience using the 
Internet. 
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“How long have you been using an Internet browser?” 


Experience 


Number of Respondents 


Percentage 


Less than one month 


4 


9% 


One month to < One year 


4 


9% 


One year to < two years 


9 


19% 


Two years or more 


29 


63% 


Total 


46 


100% 



Table 1. User Experience With a Computer Browser 



2. User Satisfaction With the Web Site 

Part II of the exit survey examined the subject’s impressions of the layout and 
content of the Web site. Questions 6 through 9 specifically targeted the subjects’ overall 
impression of the Web site. Question 6 asked, “My tour of the Web site was very 
interesting.” and the mean was 3.87 (SD=.686). The range for question 6 was 4, and the 
histogram of the frequency distribution is presented in Figure 6. Clearly many 
respondents (74%) agreed that the tour of the Web site was interesting, with only one 
person disagreeing with this statement. 



c/5 



C 

o 

CL 

c/5 

o 

Urn 

(D 

JO 

c 

o 

7 : 




Strongly Agree Neutral Disagree Strongly 
Agree Disagree 

Figure 6. “Interesting” 
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Question 7 stated, “I found the Web site easy to navigate” and the mean was 4.35, 
(SD = .822). The range for question 7 was 4, and the histogram of the frequency 
distribution is presented in Figure 7. The vast majority (91%) agreed with this statement, 
while three persons disagreed. 
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Figure 7. ‘‘Easy to Navigate’ 



Question 8 asked, “I feel the information on this Web site was in a logical form.” 
and the mean was 4.16, (SD = .556). The range for question 8 was 3, and the histogram 
of the frequency distribution is presented in Figure 8. Most respondents (91%) agreed 
that information was logically represented, with only four persons neutral on this topic. 
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Figure 8. “Web Site in Logical Form” 



Question 9 asked, “The information presented in the Web site is relevant to 
maintenance safety” and the mean was 4.37 (SD = .488). The range for question 9 was 2, 
and the histogram of the frequency distribution is presented in Figure 9. All subjects 
(100%) either strongly agreed or agreed with this statement. 
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Figure 9. “Information Relevant” 



After completing the Likert style questions in the first section of Part II, the 
respondent was asked to comment on his/her impression of the site through the open 
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ended question, “My overall impression of the Web site is;” Below are listed some of 
comments that typified the response to this question. 

“Looks like it’ll be useful. Especially the MCAS and MCAS analysis tools.” 
“Simple, straight forward and should have useful information for safety issues.” 
“Very user friendly Web site which contained both interesting and useful 
information.” 

“Should be a good tool for promoting safety and existing tools.” 

Subjects expressed general enthusiasm for the Web site and thought it would be useful in 
the distribution of safety related material. 

The next four questions (10-13) of the exit survey looked at consistency issues. 
Consistency is important in both term usage and implicit instructions, this makes 
navigation through the Web site possible with minimum confusion. Whether consistency 
is achieved was determined by the impression of the respondent. These four Likert style 
questions were posed to allow each user of the prototype to comment on their perceived 
consistency throughout the exercise. 

Question 10 asked, “During my tour of the Web site, I found the use of terms 
consistent” and the mean was 4.05 (SD = .630). The range to question 10 was 4, and the 
histogram of the frequency distribution is presented in Figure 10. The majority of users 
(74%) agreed with this statement, with only two users neutral and two disagreeing. 
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Figxare 10. “Use of Terms Consistent” 



Question 1 1 continued with consistency by asking, “During my tour of the Web 
site, I found The position of screen instructions were consistent” and the mean of this 
question was 4.31 (SD = .510). The range to question 1 1 was 3, and the histogram of the 
frequency distribution is presented in Figure 11. The majority (98%) agreed with this 
statement, and only one user was neutral. 
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Figure 11. “Screen Instmctions Consistent” 
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Question 12 asked, “I found the results obtained from the Web site were 
predictable” and the mean was 3.94 (SD = .611). The range of question 12 was 3, and the 



histogram of the frequency distribution is presented in Figure 12. A majority of the users 



(78%) found that they received consistent results from the Web site, although, nearly on 
quarter of the respondents (22%) were neutral on this aspect. 
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Figure 1 2. “Results Predictable” 



Question 1 3 asked, “During my tour of the Web site, I found adequate instructions 
were provided to navigate” and the mean was 4.22 (SD = .757). The range for question 13 
was 4, and the histogram of the frequency distribution is presented in Figure 13. Most 
users (89%) aggreed adequate instructions were provided, while three persons were 
neutral and two persons disagreed. 
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Figure 13. “Adequate Navigation Instructions” 

The next five questions on the survey (14-18) addressed the organization and 
clarity of the information presented in the prototype Web site. These questions 
concentrate on the experience of the user while learning the methods employed to 
navigate the site. An open-ended question at the end of this section provided the subject 
an opportunity to voice opinions not covered in the Likert scale questions. 

Question 14 asked, “I feel that the information on the Web site is easy to 
understand” and the mean was 4.16 (SD = .868). The range for question 14 was 4, and the 
histogram of the frequency distribution is presented in Figure 14. While a majority 
agreed (87%) the information was easy to understand, two subjects were neutral, with two 
disagreeing. 
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Figure 14. “Information Easy to Understand” 

Question 15 stated, “I feel that it was easy to learn to use this Web site” and the 
mean was 4.42 (SD = .652). The range for question 15 was 3, and the histogram of the 
frequency distribution is presented in Figure 15. A large majority (91%) agreed with this 
statement, while half (50 %) strongly agreed with the ease of learning the Web site. 
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Figure 15. “Ease of Learning the Web Site” 
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Question 16 stated, “I feel that the organization of material on this Web site is 
clear” and the mean was 4.05 (SD = .815). The range for question 16 was 4, and the 



histogram of the frequency distribution is presented in Figure 16. Most users (82%) the 



organization was clear, while five users were neutral and three users disagreed. 
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Figure 16. “Organization of Material” 



Question 1 7 stated, “I feel that I could effectively use the information on this Web 
site” and the mean was 4.35 (SD = .566). The range for question 17 was 3, and the 
histogram of the frequency distribution is presented in Figure 17. The majority of 
respondents (96%) agreed, with two persons being neutral. 
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Figure 17. “Effectively Use the Information” 



Question 18 stated, “I feel that when fully implemented, this Web site will have 
all the functions and capabilities I would expect it to have” and the mean was 4 (SD = 
.760). The range for question 18 was 4, and the histogram of the frequency distribution is 
presented in Figure 19. Most users (81%) agreed, whereas nine users disagreed. 
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Figure 18. “Web Site has all Functions I Expect” 



The open ended question in the final section of Part II of the Exit Survey asked, 
“To improve the usability of this Web site, I would.” The purpose of this question was 
twofold; first, allow the respondent to voice any usability concerns that were not covered 
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in the Likert style questions, and second, highlight any issues the study may have omitted. 



The responses to this question are summarized in Table 2. 



“To improve the usability of this Web site, I would” 


Type of Response 


Number of Responses 


Percent of Total 


No Response 


16 


35% 


Written Response 


30 


65% 



Table 2. Comments on Usability Improvements 



Of the 30 responses, most commented that the Web site was user friendly and did 
not offer suggestions for improvement to the prototype. The responses that suggested 
changes to the prototype were typified by the comments below. 



“Provide more links on the first page to make it more efficient and easier 
to the final page I am after. Index or reference at the start or bottom of the 
page would work.” 

“Perhaps on the opening slide there could be something like a table of 
contents. It would be useful to know everything that is contained on the 
Web site.” 

“... Also, home page is confusing, I would title the button START HERE 
instead of ENTER.” 

“Add a link to the NASA safety site.” 

“Use Frames.” 

A common theme of these answers was one of mild navigational preferences, some 
relating to verbage and others related to the physical layout of the Web site. 



3. User Satisfaction With the Online MCAS 

Part III of the Exit Survey contained questions pertaining to the online version of 
the MCAS. To effectively use the online MCAS, the user needed to be capable of finding 
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the MCAS in the Web site, and he/she also needed to comprehend the method of 
completing the questionnaire and submitting it. Questions 19 through 21 deal with the 
feelings about the clarity of instructions and ease of completing the online MCAS. 

Question 19 asks, “Finding the online MCAS in the Aviation Maintenance Safety 
Web site was straight forward” and the mean was 4.44 (SD = .655). The range for 
question 19 was 4, and the histogram of the frequency distribution is presented in 
Figure 19. The vast majority of users (96%) either strongly agreed or agreed with this 
statement, and two were neutral or disagreed. 
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Figure 19. “Finding the MCAS” 

Question 20 asked, “The instructions for completing the MCAS are easily 
understood” and the mean was 4.64 (SD = .488). The ramge for question 20 was 2, and 
the histogram of the frequency distribution is presented in Figure 20. All respondents 
(100%) either strongly agreed or agreed to this question. 
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Figure 20 “Instructions for MCAS Understood” 



Question 21 asked, “The task of putting answers in the MCAS was easily 
completed” and the mean was 4.55 (SD = .585). The range for question 21 was 3, and the 
histogram of the frequency distribution is presented in Figure 21. Mainly users (96%) 
strongly agreed or agreed with this question, with the exception of two persons who were 
neutral. 
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Strongly Agree Neutral Disagree Strongly 
Agree Disagree 



Figure 21 . “Submitting Answers to the MCAS was Easy” 
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The final question in Part III of the Exit Survey was the open ended question “I 
would make these changes (if any) to the administration of the online MCAS.” The 
response rate to this question is summarized in Table 3 below. 



“I would make these changes (if any) to the administration of the online 

MCAS.” 


Type of Response 


Number of Responses 


Percent of Total 


No Response 


12 


26% 


Written Response 


34 


74% 



Table 3. Comments on MCAS 



Content analysis of the open-ended responses yielded a grouping of similar 
themes. Users commented on the content of the survey as well as the administration, 
below are typical responses to the online MCAS. 



“The survey is too long... Your average trooper will not willingly (hence 
honestly) fill out such an in-depth survey.” 

“I am a Marine 53-E Pilot and many of the questions did not pertain to me. 
Some of those did not have N/A as a selection.” 

“E-6 Community- Where would it fall?” 

Administrative Comments: 

“Have an option of no graphics/cookies. Units at sea have minimal 
bandwidth and download time is terrible.” 

“I tried to tab thru [sic] the questions (instead of mouse). I thought hitting 
ENTER would record an answer, but instead... it submitted the survey 
before I was complete.” 
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having to manually scroll down continually to answer more questions 
was annoying.” 

“Have more questions on one page” 

Comments in this area ranged from a concern that the length of the survey might 
inhibit the validity of some respondent’s answers (attention span) to physical 
limitations of the movement through the Web site (utilizing back button of 
browser, etc.). 

The final area of the Exit Survey was labeled “Comments” and posed two the 
open ended questions “The most negative aspects of this Web site were” and “The most 
positive aspects of this Web site were”. The response rate to each of these questions is 
summarized in Table 4 and Table 5. 



“The most negative aspects of this Web site were.” 


Type of Response 


Number of Responses 


Percent of Total 


No Response 


5 


11% 


Written Response 


41 


89% 



Table 4. Comments on Negative and Positive Aspects of online prototype 



“The most positive aspects of this Web site were.” 


Type of Response 


Number of Responses 


Percent of Total 


No Response 


3 


7% 


Written Response 


43 


93% 



Table 5. Comments on Negative and Positive Aspects of online prototype 
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The Comments on both negative and positive aspects of the prototype Web site 
were analyzed and the most representative comments are listed below. 

Negative Aspects: 

“Having to back out after completing the survey” 

“Marine billets not included” 

“Lack of visual stimulation for the Gen X/ Mountain Dew crowd.” 

“Too much wasted space on the main pages” 

“Survey might be too long, you’ll surpass the attention span of your target 
audience.” 

“Survey on the long side” 

Positive Aspects: 

“Ease of use, simplicity” 

“Power point Lectures” 

“A great information hwy for maintenance. Nothing but positive results 
can come from this site.” 

“Generally easy to use” 

“Easy access, easy to navigate, accessible to everybody” 
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“Should provide good background data” 



The negative comments centered on a concern that young sailors would not take 
the time to complete the survey because of it’s length. The positive comments 
characterized a Web site that was easy to use and contained useful information for 
safety programs. 

B. ANOVA TESTING 

1. One Way ANOVA 

SPSS software provides a metric called the significance level, also known as p 
value, which allows the researcher to interpret a calculated statistic in the generated 
output without using a critical value table. The symbol p equates to the probability of a 
Type I error. A Type I error is the likelihood of rejecting the null hypothesis when it is 
actually true. If the reported p is less than .05, the finding is statistically significant and 
the null Hypothesis (no difference) is rejected (Morgan & Griego, 1998). 

A one-way ANOVA was used to compare the means of different respondent 
experience levels to test for significant differences in levels of usability. ANOVA 
assumes the Dependant Variable (DV) is interval scale, normally distributed, and the 
variance of the groups are equal. If the assumptions of ANOVA are violated, an 
alternative nonparametric test called the Kruskal-Wallis (K-W) test should be used to 
compare the differences. It was not necessary to use the Kruskal-Wallis test with the data 
from this study, since it met all the parameters necessary to for ANOVA analysis. The 
Independent Variable (IV) in this study was represented as ordinal data while the DVs 
were measured on an interval scale. The distributions of each of the DVs were compared 
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to a normal distribution curve to ensure they approximated the normal distribution. 

The equality of variances was tested using the Levene Test (Kotz, Johnson, & 
Read, 1 988). Since the Levene statistic was not significant (p >.05), the assumption of 
equal variances was not violated and the K-W test was not needed. (Morgan & Griego, 
1998). The ANOVA statistic compares means between groups and is used to identify a 
statistically significant difference between means. If the p value calculated by an 
ANOVA test is significant (p < .05) then there is a significant difference between groups 
with respect to the DV. In order to answer the difference between the IV and the 
individual DVs, hypotheses were developed and tested. 

2. Formulation of The Null Hypothesis 

The null hypothesis is used for testing; it is a statement that no difference exists 
between the parameter and the statistic being compared to it (Cooper & Emory, 1995). 
The Alternative hypothesis is the logical opposite of the null hypothesis. If statistically 
significant, the null hypothesis can be rejected, and the alternative hypothesis is accepted. 
The following hypotheses formed and tested to compare how the IV (User experience 
level) impacted the DV (User impressions). 

1 ) rV: How long have you been using an Internet Browser? 

DV; I found the Web site easy to Navigate. 

NULL HYPOTHESIS: there is no difference in how experience level impacted 
the ease of navigation of the Web site. 

The F statistic with 3, and 42 degrees of freedom was 1.494. This resulted in a p value of 
.234, which did not allow rejection of the null hypothesis. 



51 



2) rv : How long have you been using an Internet Browser? 

DV: I feel the information on the Web site was in a logical form. 

NULL HYPOTHESIS: There is no difference in how experience level impacted 
perception of whether the information was presented in a logical form. 

The F statistic with 3, and 42 degrees of freedom was .799. This resulted in a p value of 
.503, which did not allow rejection of the null hypothesis. 

3) IV : How long have you been using an Internet Browser? 

DV: I found that adequate instructions were provided to navigate the Web site. 
NULL HYPOTHESIS: There was no difference in how experience level impacted 
whether respondent felt enough instructions were provided. 

The F statistic with 3, and 42 degrees of freedom was 1.494. This resulted in a p value of 
.234, which did not allow rejection of the null hypothesis. 

4) rv : How long have you been using an Internet Browser? 

DV: It was easy to learn to use this Web site. 

NULL HYPOTHESIS: There is no difference in how experience level impacted 
the ease of learning to use the Web site. 

The F statistic with 3, and 42 degrees of freedom was 2.287. This resulted in a p value of 
.097, which did not allow rejection of the null hypothesis. 

With each case, the null hypothesis could not be rejected. The comparison of the 
subject’s experience level with Internet browsers to usability concerns, does not impact 
the impression the subject had on the Web site’s usability. 
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V. CONCLUSION 



A. SUMMARY 

Naval Aviation is posed with the challenge of reducing its mishap rate, making it 
essential to pursue programs and processes to achieve this. Organizational climate has 
been linked to risk management and, consequently, experienced mishaps. Understanding 
the climate that pervades an organization is essential if changes are to be made to 
ultimately effect the mishap rate. 

Many theoretical frameworks have been developed to understand organizational 
climate; one that fits the characteristics of Naval Aviation is the High Reliability 
Organization model (HRO). Working within the HRO framework, the Command Safety 
Assessment (CSA) survey evaluated the safety climate within Naval Aviation Squadrons 
from aircrew perspective. The Maintenance Climate Assessment Survey (MCAS) was 
designed to measure HRO characteristics, but from a perspective of the aviation 
maintainer. 

In order to provide for an effective vehicle to distribute the MCAS, the World 
Wide Web (WWW) was explored. The WWW offers an opportunity to obtain a wide 
distribution of material at a very small, fixed cost. Shrinking budgets within the military 
necessitates finding new methods of mission accomplishment, therefore making the 
efficiency of the WWW a method of allowing process improvement within the confines 
of shrinking budgets. 
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B. CONCLUSIONS 



This research, based on the initial research questions, has given the following: 

1) Is there a need to disseminate and administer Web based materials to aviation 
units? Naval Aviation is challenged to meet aggressive reductions in its mishap rate by 
the year 2000. Even with the record low mishap rates that have been achieved in recent 
years, continued improvement is necessary. The HFQMB is seeking the “invention” of 
new processes and procedures to help meet the goals set forth in mishap reduction. A 
Web based MCAS will further facilitate climate assessment and evaluation throughout 
the Naval Aviation Fleet, improved methods of assessment will lead to process 
improvements for Naval Aviation. 

2) Do the U.S. Navy and Marine Corps squadrons have the necessary 
software/hardware capabilities to utilize the WWW? As technology permeates the fleet, 
it is clear that the ability to utilize the WWW at the squadron level exists. A total of 97 
percent of the subjects in this study had access to the WWW (n = 46). The remaining 3 
percent involved in this study were not familiar with the computing capability at their 
future squadron assignment. 

3) Can the MCAS be effectively distributed and administered using the WWW? 
Development software available enables the design and implementation of surveys for 
applications on the WWW. The HTML editors that are commonly available facilitate the 
design of a Web site for distribution of the MCAS. The current level of technology can 
easily support the dissemination and collection of data from the MCAS. 
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4) Once distributed, do personnel have the necessary ability to utilize the 
electronic version of the MCAS? The subjects in this study were representative of ASOs 
throughout the fleet, 82 percent (n = 46) had over one year experience using the Internet. 
Successfully navigating the prototype of the Naval Aviation Maintenance Safety Web 
Site, subjects not only demonstrated the requisite knowledge for using the online tool, 
they demonstrated an acceptance and enthusiasm for the project. 

Adequate knowledge exists within the ASO community to make effective use of 
the WWW as a distribution method for the MCAS. Using the efficiency obtained 
through the WWW allows increased distribution of the MCAS with minimal cost. 
Taking advantage of the software and hardware that is currently in place within aviation 
squadrons is an additional step to achieving the improvements that Naval Aviation must 
accomplish to meet its goals of mishap reduction. 

C. RECOMMENDATIONS 

Many areas remain open for further development and testing of the Naval 
Aviation Maintenance Safety Web Site. It is suggested that further research efforts 
include the following areas: 1) Inclusion of survey feedback into the layout and design of 
the Web site, 2) Populating the pages that were data space holders, (e.g., summaries of 
mishaps, etc.), and 3) Further development of data capture methods. 
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APPENDIX A. AVIATION MAINTENANCE SAFETY WEB SITE 



This appendix consists of three parts: (1) A verbal description of the 

pages that make up the Aviation Maintenance Safety Web site, (2) A symbolic 
depiction of the logical layout of the Web site, (3) Screen captures of the pages 
that make up the complete prototype Aviation Maintenance Web site. 

Part 1. Verbal Description of Web Pages 
Page 1 : Title page for the Web site. 

Page 2: Welcome/Purpose page. Branches to page: 2.1 

Page 2.1: Safety Survey Page. Branches to pages: 2.1.1, 2.1.2, and 2.1.3 

Page 2.1.1; MC AS 

Page 2.1.2: Fleet Statistics on Excel 

Page 2.1.3: Interpretation Materials 

Page 2.2: Related Web sites. 

Page 3: Aviation Maintenance Training Page. Branches to pages: 3.1, 3.2 

Page 3.1: Groundcrew Coordination Training. Branches to pages: 3.1.1, 3.1.2, and 3.1.3 

Page 3.1.1; Workshop Materials. 

Page 3.1.2: Student Guide. 

Page 3.1.3: Case Studies. 

Page 3.2: FAA Document. 

Page 4: Community Maintenance Human Factors Briefs. 

Page 4.1 : Community Human Factors Briefs for Jets. 
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Page 4.2: Community Human Factors Briefs for Props. 
Page 4.3: Community Human Factors Briefs for Helos. 
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Part 2. Symbolic Depiction of the Logical Layout of the Web Site 
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Part 3. Screen Captures of the Aviation Maintenance Safety Web Site 
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APPENDIX B. INSTRUCTION SHEET FOR SUBJECTS 



Instructions for viewing the 

Aviation Maintenance Safety Web site 

These instructions will assist in viewing the Aviation Maintenance Safety Web site. 



Computers to use: 

In the ASO Ready Room Deck 
Computer names: 

In the Computer Lab Deck 

Computer names: 



Prowler 

Harrier 

Skywarrior 

Texan2 

Viking 

Cobra 



Accessing the Web site: 

1 . Log on the computer using your group ASO account 

2. On the desktop, double click on the icon named Aviation Maintenance Safety Web 
site 



^viaboa - 

^VIOnsrlCQ! 



3. When the initial page of the Web site opens, utilize the function buttons within 
Microsoft Internet Explorer (Back, Forward, Stop, etc.) to “surf’ through the Web site. 
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APPENDIX C. TASK LIST 



Aviation Maintenance Safety Web Site 
And 

Online Version of the Maintenance Climate Assessment Survey (MCAS) 



Purpose: The following tasks are designed to introduce a developmental version of the 
Aviation Maintenance Safety Web Site. This Web site is being created to assist aviation 
squadron safety personnel in assessing the climate of safety within their organization, and 
to provide useful safety reference material for squadron training. 

Your input will be used to improve this Web site and maximize it's utility to the squadron 
safety officer. 

Directions: The following list of tasks will guide your "surf of the Aviation 
Maintenance Safety Web Site. Upon completion of the tasks, a short questionnaire will 
query your experiences with the site. There is room for comments on the survey, feel free 
to make suggestions. 

Your Assistance in making this future Web site productive is greatly appreciated! 



TASK LIST 

As you complete the following tasks, make a mental note 
of any difficulties you may experience. 

1 . During your tour of the site, find the online version of the Maintenance Climate 
Assessment Survey (MCAS), complete the survey and submit your results using the 
submit button at the bottom of the form 



2. Find the Naval Safety Center homepage through the Links provided in the Web site 

3. Search the Web site as if you were planning squadron training; find the portion of the 
Web Site that has Microsoft PowerPoint presentations to download. 

4. Find the area of the Web site where mishaps by aircraft community are located. 
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APPENDIX D. EXIT SURVEY 



User’s impression of the 
Maintenance Safety Web site and 

Maintenance Climate Assessment Survey (MCAS) 



Purpose: This survey evaluates a user's overall satisfaction of the Aviation Maintenance Safety Web site, it consists of 
three parts. 

Part I: Demographic Information. This information is used to establish the user's computer experience and determine 
the sofhvare/hardware being used in the fleet. 

Part II: User satisfaction with the Aviation Maintenance Safety Web site. This Section deals directly with participant 
experience as they browsed the prototype Maintenance Safety Web site. 

Part III: User Satisfaction with the online version of the Maintenance Climate Assessment Surv'ey (MCAS). This 
section allows participants to comment on the online MCAS. 



Part I. Demographic Information 

1 . I am attached to a command that performs maintenance at the: 

(Choose one from the list) 

Squadron Level 
Intermediate Level 
Depot Level 
None of the Above 

2. How long have you been using an Internet Browser? 

(Choose one from the list) 

I Less than one month 

□ 1 month to less than 1 year 

□ 1 year to less than 2 years 

Z 2 years or more 

3. What Internet Browser do you normally use? 

(Select all that apply from both columns) 

Netscape Navigator 
I Microsoft Internet Explorer 

Z AOL's version of Internet Explorer 

Z Other Browser 

Z None 



At Work I use: At Home I use: 

Z Netscape Navigator Z 

Z Microsoft Internet Explorer 

Z AOL’s version of Internet Explorer 

Z Other Browser 

Z None 



4. W'liat categories of software have you used and are familiar with? 
(Select all that apply) 

Z Word Processor (MS Word, WordPerfect, etc.) 

D Spreadsheet (MS Excel, Lotus 123, etc.) 
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lJ Graphic Software (Corel Draw, Adobe Photoshop, etc.) 

□ Electronic Mail (AOL, Eudora, etc.) 

5. What computer systems do you use? 

(Select all that apply) 

H WindowsAVindows NT 

I Macintosh 

□ UNIX 

□ Other 



PART 11 



USER IMPRESSIONS 

(Fill in the choice that matches your impression) 



My tour of the Web site 
Was very interesting 


Strongly 

Agree 

□ 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


I found the Web site 
easy to Navigate 




- 


- 


P 


r 


I feel the information on 
the Web site was in a 
logical form 


- 


- 


□ 




p 


The information presented 
in the Web site is relevant 
to maintenance safety 






- 


□ 


- 


My overall impression of the Web site is: 























During my tour of the Web site, 1 found: 










Strongly 


Agree 


Neutral 


Disagree 


Strongly 


Agree 








Disagree 


The use of terms consistent [I 




□ 


□ 


□ 



82 



The position of screen 
instructions were consistent 



The results obtained from 

the Web site were predictable I- 


3 


- 


2 


- 


That adequate instructions 
were provided to navigate 
the Web site C 


_ 


1 






1 feel that: 










The information on the 
Web site is easy to 
Understand Z 


r 


— 


_ 


r— 


It was easy to learn to 

use this Web site Z 


z 


- 






The organization of material 
on this Web site is clear Z 


- 


- 


□ 


- 


I could effectively use the 
information on this Web site Z 


z 


- 




I— 


When fully implemented, this 
Web site will have all the 
functions and capabilities I 
would expect it to have Z 


- 


3 


□ 


- 



To improve the usability of this Web site, I would: 
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Part III 



Online MCAS 



Strongly Agree Neutral Disagree Strongly 

Agree Disagree 



Finding the online MCAS 
in the Aviation Maintenance 
Safety Web site was 
straight forward 



The instructions for 
completing the MCAS 

are easily understood I □ 



The task of putting answers 
in the MCAS form was 
easily completed 



I would make these changes (if any) to the administration of the online MCAS 



Comments 

The most negative aspects of this Web site were: 



The most positive aspects of this Web site were: 



Your participation is appreciated 
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